1. Field of the Invention
The present invention relates to general software testing, and more particularly to combinatorial test generation for compatibility testing of Java technologies.
2. Description of the Related Art
Currently, Java environments can be categorized into various Java technologies. A Java technology is defined as a Java specification and its reference implementation. Examples of Java technologies are Java 2 Standard Edition (J2SE), Java 2 Enterprise Edition (J2EE), and Mobile Information Device Profile (MIDP). As with most other types of Java software, a new Java technology should be tested to assure consistency across multiple platforms. This testing is generally performed using compatibility testing.
Compatibility testing refers to the methods used to test an implementation of a Java technology specification in order to assure consistency across multiple hardware platforms, operating systems, and other implementations of the same Java technology specification. When this assurance is accomplished by means of a formal process, application developers can then be confident that an application will run in a consistent manner across all tested implementations of the same Java technology specification. This consistent specification-based behavior is a primary function of compatibility testing.
The process of developing compatibility tests for a fairly large API is usually a multi-month process that includes the development of a number of related components. Unlike product testing, which can begin as soon as some part of the program is written, compatibility testing requires both a working implementation and an accurate, complete specification in order to complete the test development process.
FIG. 1 is a flow diagram showing a prior art process 100 for creating compatibility tests. As shown in process 100, a Java specification 102 is processed to create a set of relatively small testable assertions 104. Statements are generally considered testable assertions if they are intended to describe behavior of an API and can be tested by compatibility tests. Also, examples or sample code pieces that are provided in the specification are typically testable and can be verified by the compatibility tests. In this sense, examples or sample code are generally considered testable assertions.
On the other hand, statements intended to describe the behavior of an API, but which cannot be tested by compatibility tests due to the special nature of the behavior or functionality, are generally considered non-testable assertions. Similarly, some statements form general descriptions of the API such as a description of a package, class, method, or field, and so forth. If such a general description does not describe behavior, but is aimed rather at providing a context for the rest of the text, then such a statement generally is not intended to be an assertion. Hence, these statements are generally not considered to be assertions due to their nature.
Once the set of testable assertions 104 is obtained, one or more tests 106 for each assertion is written. Unfortunately, it is not always clear in conventional compatibility testing exactly what should be tested in each assertion and how many tests should be written for each assertion. Theoretically, every logical entity in the assertion could be tested and an infinite number of tests could be written to fully test the assertion. But in practice this, of course, is not affordable.
In view of the foregoing, there is a need for methods providing improved compatibility test generation. The methods should generate compatibility tests that utilize the least number of combinations of assertion entities provide an acceptable quality of testing of the assertion.